Evidence Of External Validity Research Articles

Hypertrophic cardiomyopathy detection with artificial intelligence electrocardiography in international cohorts: an external validation study

Abstract Aim Recently, deep-learning artificial intelligence (AI) models have been trained to detect cardiovascular conditions, including hypertrophic cardiomyopathy (HCM), from the 12-lead electrocardiogram (ECG). In this external validation study, we sought to assess the performance of an AI-ECG algorithm for detecting HCM in diverse international cohorts. Methods and Results A convolutional neural network-based AI-ECG algorithm was developed previously in a single-center North American HCM cohort (Mayo Clinic). This algorithm was applied to the raw 12-lead ECG data of HCM patients and non-HCM controls from 3 external cohorts (Bern, Switzerland; Oxford, UK; and Seoul, South Korea). The algorithm’s ability to distinguish HCM vs non-HCM status from the ECG alone was examined. A total of 773 HCM patients and 3,867 non-HCM controls were included across 3 sites in the merged external validation cohort. The HCM study sample comprised 54.6% East Asian, 43.2% White, and 2.2% Black patients. Median AI-ECG probabilities of HCM were 85% for HCM patients and 0.3% for controls (p < 0.001). Overall, the AI-ECG algorithm had an AUC of 0.922 (95% CI 0.910-0.934), with diagnostic accuracy 86.9%, sensitivity 82.8% and specificity 87.7% for HCM detection. In age- and sex-matched analysis (case-control ratio 1:2), the AUC was 0.921 (95% CI 0.909-0.934) with accuracy 88.5%, sensitivity 82.8%, and specificity 90.4%. Conclusion The AI-ECG algorithm determined HCM status from the 12-lead ECG with high accuracy in diverse international cohorts, providing evidence for external validity. The value of this algorithm in improving HCM detection in clinical practice and screening settings requires prospective evaluation.

Development and validation of the Driver Attention Regulation Scale: A measure of the perceived ability to regulation attention on the road

The ability to regulate attention while driving is a vital skill for remaining safe on the road. However, there is no current self-report method that measures such a construct, potentially limiting research in the area of attention and distraction. This study aimed to develop and validate a new measure, the Driver Attention Regulation Scale (DARS), that can be used to identify how drivers perceive their ability to regulate attention while driving. To achieve this aim, two online surveys were shared with unique samples of Australian drivers. Following the development of an initial item pool, principal components analysis and follow up confirmatory factor analysis were used to test the structure of the items and revealed that three factors emerged: attentional presence (e.g., focus; vigilance), attentional flexibility (e.g., awareness; shifting), and attentional capacity (e.g., processing speed; cognitive resources). Each of the subscales (α = 0.75–0.88) and the total scale (α = 0.87–0.91) exhibited a good internal consistency across the samples. Evidence of external validity was also demonstrated, as the DARS was positively associated with generalised attention regulation measures, and driver confidence, but negatively associated with emotion regulation difficulties, driver stress, and driving errors/lapses. Mediation analyses also indicated that the attentional constructs measured by the DARS may indirectly influence driving behaviours, via their effect on driver stress and confidence. Overall, the DARS has demonstrated promise as a new measure of driving-related attentional skills that can be used in future research to better understand how situational factors may impact drivers’ confidence, and also how confidence may influence driving behaviours (e.g., speeding).

Measuring mental health action competencies in school teachers: internal and external validity evidence.

Mental health literacy is receiving increasing research attention due to growing concerns for mental health globally. Among children, teachers have recently been recognized as playing a vital role in the recognition and reporting of potential mental health issues. A nationally sampled cross-section of teachers was surveyed to examine the discriminant validity of the mental health literacy measure across levels of teaching. A survey collected a total of n = 369 teacher responses in Switzerland (Kindergarten = 76, Primary = 210, Secondary = 83). Item response theory (IRT) analyses were conducted. Inspection of psychometric properties indicated removal of two weak performing items. The 15-item measure exhibited a significant mean difference, such that class-responsibility function scored higher (M = 2.86, SD = .45) than non-responsible function (M = 2.68, SD = .45) teachers [t(309) = -2.20, p = .01]. It also exhibited a significant mean difference, such that more subjective experienced scored higher (M = 2.86, SD = .45) than less subjective experienced (M = 2.68, SD = .45) teachers [t(210) = -8.66, p < .01]. Hypotheses regarding age and role tenure were in the expected direction, but non-significant. The MHL measure for teachers demonstrated sound measurement properties supporting usage across teaching levels.

Psychometric Evaluation of the School Climate and School Identification Measure-Student on Chilean Students: A Bifactor Model Approach.

School climate is a relevant construct for understanding social relations at school. The SCASIM-St has been widely defined as a multidimensional construct; however, new factor structures have not been explored through evidence that allows for interpreting school climate scores from an approach that respects the multidimensionality of the scale and, at the same time, allows for identifying the degree of essential unidimensionality in the data. Consequently, the objective was to analyze the psychometric properties of the SCASIM-St from a bifactor model approach, evaluating the influence of a general school climate factor versus five specific factors. The study involved 1860 students of both sexes (42% males and 58% females), with an average age of 16.63 years (SD = 0.664), from 17 secondary schools in Chile. The results obtained by a confirmatory factor analysis provided evidence that the best model was the bifactor model for the 38 items, with one general factor and five specific factors. The Explained Common Variance (ECV) values and reliability levels by hierarchical omega accounted for a strong general school climate factor with high levels of reliability. Evidence of external criterion validity, assessed through the attitude toward authority scale (AIA-A), showed a theoretically expected and significant relationship between the factors of both instruments. This study confirmed the psychometric robustness of the SCASIM-St scale by means of a bifactor model, allowing for a new, essentially unidimensional interpretation of the scale scores and providing an instrument to measure school climate in Chile.

Psychometric Validation of the Tactical Assessment Instrument in Football for Use in Physical Education and with Youth Sport Teams.

Recently, the Tactical Assessment Instrument in Football (TAIS) was developed to address football's tactical requirements in terms of game phases, learners' roles, and organizational levels of play. In this study, we sought to evaluate the psychometric properties of this existing instrument in the Physical Education (PE) context and for youth sport teams (ages 8-12years). Our research design involved: (a) participants playing 37 football games, (b) assessing participants via observation with the TAIS and (c) conducting psychometric analysis on the obtained assessment data. For the psychometric analysis, we carried out our work in four phases: (a) a statistical analysis of the evaluation criteria, (b) analysis of the instrument's structural dimensions, (c) internal item reliability analysis, and (d) provision of evidence for external validity. Participants were 592 children (156 girls, 436 boys; ages 8-12years) from 74 PE classes or sport teams (from three institutional contexts - schools, community-based sports, and sports clubs) in Spain. We selected 12 tactical criteria and grouped them into four theoretical dimensions (appropriate attack, inappropriate attack, appropriate defense, inappropriate defense) to confirm the instrument's structural dimensions. The Cronbach alpha and Omega McDonald coefficients were greater than .70. The TAIS discriminated between participants from the three institutional contexts, offering evidence of the instrument's external validity. Thus, the TAIS now has sufficient psychometric support for assessing learning of football tactics in PE and youth sport teams. This instrument facilitates an ecological assessment of youth players' understanding of football tactics through four theoretical tactical dimensions and 12 criteria involving information about four learners' roles (both attack and defense, on- and off-the-ball) and three organizational tactical levels of play (whole team, small groups of learners, and individual learners).

Cultural adaptation to Spanish and psychometric analysis of the Woman Abuse Screening Tool (WAST) in Peruvian adult women

BACKGROUND: Partner violence is a public health problem. In Peru, women with spouses are vulnerable to suffering it. The Woman Abuse Screening Tool (WAST) is a widely recommended instrument for the early identification of these cases. Therefore, our objective was to adapt and analyse the psychometric properties of the WAST scale in adult Peruvian women. METHODOLOGY: This was an instrumental study, with a sample of 628 women who were in a marital condition of cohabitation or married at the time of the study. The mean age was 35 years (M = 35.58; SD = 10.92) and on average they were living with their partner for 9 years (M = 9.93; SD = 8.41). This sample was selected by non-probabilistic snowball sampling. The Woman Abuse Screening Tool (WAST), Generalized Anxiety Disorder-7 (GAD-7) and Patient Health Questionnaire (PHQ-9) were used. The validity of the internal structure was verified with confirmatory factor analysis (CFA), convergent validity by correlation coefficients between the WAST, GAD-7 and PHQ-9, while reliability was verified using Cronbach’s alpha and McDonald’s Omega coefficient. RESULTS: Factorial analysis indicated that the one-dimensional model showed adequate goodness-of-fit indices (CFI = 0.998; TLI=0.991; RMSEA=0.103; SRMR=0.075), and reliability coefficients (α = 0.899 and ω = 0.916) were optimal. Additionally, the WAST showed strong external validity based on the relationship with the PHQ-9 and GAD-7 scores (Rho &gt; 0.50). CONCLUSIONS: The Woman Abuse Screening Tool (WAST) scale demonstrated solid evidence of internal and external validity and optimal reliability in adult Peruvian women. Its use will benefit the early identification of victims of intimate partner violence in primary health care and further research.

Development of the PREMIUM computerized adaptive testing for measuring the access and care coordination for patients with severe mental illness

Severe mental illness (SMI) patients often have complex health needs, which makes it difficult to access and coordinate their care. This study aimed to develop a computerized adaptive testing (CAT) tool, PREMIUM CAT-ACC, to measure SMI patients' experience with access and care coordination. This multicenter and cross-sectional study included 496 adult in- and out-patients with SMI (i.e., schizophrenia, bipolar disorder, or major depressive disorder). Psychometric analysis of the 13-item bank showed adequate properties, with preliminary evidence of external validity and no substantial differential item functioning for sex, age, care setting, and diagnosis, making it suitable for CAT administration. A post-hoc CAT simulation demonstrated that the tool was efficient and accurate, with an average of seven items, compared to the full item bank administration. Its use by clinicians can contribute to optimizing patient care pathways and transitioning towards more person-centered healthcare.

The validity and reliability of the Turkish version of revised metacognitive awareness of reading strategies inventory

This study aims to adapt the revised Metacognitive Awareness of Reading Strategies Inventory (MARSI- R) into Turkish. MARSI-R is a self-report instrument designed to assess students’ metacognitive awareness of reading strategies and perceived strategy use when reading school-related materials. 525 students (65% female, 35% male, Mage = 13 years old.) from multiple school types and degrees participated in this study. A stepwise validation procedure was used to translate and produce a Turkish version of the inventory. Evidence of structural and external aspects of validity for the inventory was collected. The 15-item inventory had a three-factor solution (global reading strategies, problem-solving strategies, and support reading strategies), as supported by confirmatory factor analysis. Turkish version scores were positively correlated with students' perceived reading ability, which provides evidence of MARSI-R's external validity. The coefficient of stability was calculated using data from 85 students who took the Turkish version of the MARSI-R twice in a five-week interval. The study’s overall results provided evidence of the reliability and validity of the inventory. According to the results presented in this study, the Turkish version of the inventory can be implemented to assess the students’ metacognitive awareness of reading strategies and perceived strategy use. The findings show that the adapted inventory can be used to obtain valid and reliable results for Turkish lower and upper secondary school students.

The Self-Efficacy to Regulate Physical Activity Scale: Exploring the Factor Structure, Temporal Invariance, and External Validity in a Latent Variable Framework

ABSTRACT The purpose of this study was to explore the validity of evidence for self-efficacy to regulate physical activity scale (SERPA) measurement using an exploratory latent variable approach. The objectives were to explore the dimensionality, temporal invariance, and external validity of scores produced by the SERPA, a modified version of the barriers self-efficacy scale. Data (N baseline = 461 and N 30 days post-baseline  = 427) from the Well-Being and Physical Activity Study (ClinicalTrials.gov, identifier: NCT03194854), which deployed the Fun For Wellness intervention, were analyzed. A two-dimensional factor structure explained responses to the SERPA at baseline. There was evidence for at least partial strict temporal measurement invariance for this two-dimensional structure. There was evidence for external validity of the SERPA measurement as evidenced by group assignment (i.e. the intervention) exerting a direct effect on the proposed two-dimensional structure of latent self-efficacy to regulate physical activity in adults with obesity at 30 days post-baseline.

Validation of the Violent Ideations Scale (VIS) in Spain.

Research has shown that violent ideations (VIs) may play a key role in aggression and violence. However, there is no tool to measure this construct adapted to the Spanish language so far. The current study aims to translate and evaluate the psychometric properties of the Violent Ideations Scale (VIS) in European Spanish. This study counts with a convenience sample of 752 native Spaniards or Spanish speakers residing in Spain, between 18- and 75-years old, 53% identified as female and 47% identified as male. The dimensionality, gender measurement invariance, concurrent validity and reliability of scores were assessed. Initial evidence of external validity was collected. Results confirmed the unidimensionality of the instrument. Measurement invariance held across females and males and reliabilities were high. VIS-ES scores correlated with aggressive behavior during the previous month. The VIS-ES seems suitable for use in non-clinical Spanish speaking samples.

Short-term and long-term survival in patients with prevalent haemodialysis—an integrated prognostic model: external validation

ObjectivesPrognostic tools with evidence for external validity in routine clinical practice are needed to align care with patients’ preferences and deliver timely supportive services. Current models have limited, if any,...

Learning Approaches: Cross-Cultural Differences (Spain-Argentina) and Academic Achievement in College Students.

Learning approaches describe the students' degree of cognitive commitment to learning in diverse types of academic tasks and educational environments. Even though from a micro-level perspective different profiles of approaches have been identified in high-achievement undergraduates attending several majors, such profiles have not been examined from a macro-level approach in terms of distinct educational cultures. Therefore, the research involved two studies conducted on undergraduates from Argentina and Spain: The first one was aimed at analyzing the psychometric features of the Approaches and Study Skills Inventory for Students (ASSIST) whereas the second was focused on examining the learning approaches profiles of high and low achievers attending the same major (Psychology) in two different educational cultures (Spain and Argentina). The scale's original internal structure, examined on a sample of 400 participants (50% Spanish), was verified except for one item, which was fatherly eliminated. The resulting structure was tested and proven verified in a new sample (N = 1,334; 58.3% Spanish) by confirmatory factor analysis, factorial invariance, and internal consistency studies. External validity evidence was examined as well. Additionally, norms to be used in the professional field were calculated.Profiles of learning approaches by academic achievement from each country were examined by latent class analysis. In both cases, high achievers reported higher and more frequent use of the Deep and Strategic approaches and lower and less frequent usage of the Surface one. Further studies should replicate these analyses in undergraduates attending other majors in order to test the hypothesis sustaining these findings' generalization.

The Presentation of Interest Scale on Reflective Thinking and its Internal and External Validity

Studies on prediction recognize the presence of a large number of predictors of school-academic performance, as well as the existence of well-established predictors, nevertheless some psychoeducational variables remain little explored, even though their great potential. One of these is reflective thinking. The Laboratory for Research on Cognitive Architecture (LAICO) at the Federal University of Minas Gerais brought forth a contribution to reflective thinking. This is the development of the Interest Scale on Reflective Thinking which measures the interest of people in thinking reflectively. In this paper we present the Interest Scale on Reflective Thinking where it is shown in detail its structure and content. We also show evidence of structural validity, external validity and reliability of the scale.

Psychometric Properties of the Sport Multidimensional Perfectionism Scale-2 in Czech Adolescent Athletes: An Exploratory Approach

This study presents a thorough examination of the psychometric properties of the Czech version of the Sport Multidimensional Perfectionism Scale-2 in adolescent athletes. Factorial structure, external validity, and internal consistency were evaluated using a cross-sectional design. In addition, three-month temporal stability was also assessed. The cross-sectional study sample consisted of 243 respondents (50.2% females; mean age 16.40 years). Factorial structure of the Sport-MPS-2 was examined using exploratory factor analysis, evidence for external validity was based on bivariate and canonical correlations with Big Five Inventory-2 (short) and Sport Anxiety Scale-2. The longitudinal part of the study was performed on a subsample of 96 participants. Factorial structure in our sample fully corresponded to the structure suggested by the authors of the instrument. Internal consistencies of subscales were satisfactory. Regarding external validity, Doubts about Actions and Concern over Mistakes were positively related to sport anxiety, and positive relationship was found also between Concern over Mistakes and Negative Emotionality. No substantial relationships were found in case of subscales constituting perfectionistic strivings. Temporal stability of Sport-MPS-2 over three months was satisfactory. The Czech version of Sport-MPS-2 can be considered a reliable instrument for assessing perfectionism in sport in the adolescent population.

Study of the Psychometric Properties of the Social Self-Efficacy Scale with Spanish Adolescents by Gender, Age and Family Socioeconomic Level.

Social self-efficacy has been shown to be a key resource for adolescents’ social experiences with peers and a predictor of prosocial behaviour among adolescents. However, differences by gender, age and socioeconomic level have previously been found in social self-efficacy. The objective of this study is to assess the psychometric properties of the subscale of social self-efficacy from the Self-Efficacy Questionnaire for Children (SEQ-C) developed by Muris (2001) in a representative sample of Spanish adolescents while considering gender, age and socioeconomic level differences. In general, the results showed good psychometric properties and a one-dimensional structure with high internal consistency, adequate explained variance and evidence of external validity for the subscale. Furthermore, the invariance analysis demonstrated that the social self-efficacy subscale shows no bias when used with populations of adolescents who differ by gender, age and socioeconomic level. The results indicate that the Spanish version of the social self-efficacy subscale of the SEQ-C is an adequate measurement instrument for assessing adolescents’ perception of their own social skills.

The Spanish adaptation of the relationship power inventory

Power dynamics are fundamental when negotiating conflicts. However, no instrument for measuring power in romantic relationships has been adequately adapted to Spanish culture. The goal of this research was to adapt the Relationship Power Inventory (RPI; Farrell et al., 2015) to Spanish culture and language, filling this gap by providing a rigorous instrument for evaluating this construct. Study 1 was conducted to obtain evidence based on Spanish adaptation of RPI content. Once the Spanish adaptation of the RPI was built and we obtained validity evidence based on the test content, in Study 2, the scale was administered to two different samples of the adult population following a cross-validation approach. Specifically, in Sample 1 (N = 400), the training sample, a statistical analysis and an exploration of the dimensional structure and reliability of the measure were carried out. In Sample 2 (N = 755), the validation sample, the internal structure of the scale was confirmed, and evidence of external validity and generalization was obtained. The exploratory and confirmatory factor analysis showed a good fit for the four-factor structure. These dimensions were invariant to gender and had adequate validity based on their relationship with other variables (dependence on the partner, conflict-resolution strategies, and psychological well-being). In sum, the Spanish version of the RPI (SARPI) is a reliable instrument with sufficient valid evidence to provide accurate measurement of power differences in the context of romantic relationships.

Psychometric properties of a workplace spirituality measure

Orientation: Scholars are encouraged to provide contextualised definitions of spirituality (e.g. workplace spirituality) and validate measures using non-academic samples and advanced statistics.Research purpose: This study aimed to determine (1) the dimensionality associated with a measure of workplace spirituality and (2) whether the estimated primary factor scores (alignment with organisational values, meaningful work, sense of community) outperforms the prediction of the estimated scores for perceived employee performance compared with the estimated scores for the general factor (workplace spirituality).Motivation for the study: Valid measures of workplace spirituality are required for conducting research to advance our understanding of its relationship with organisational outcomes.Research approach/design and method: Using a cross-sectional survey design, 789 public servants completed measures on workplace spirituality and perceived employee performance. A three-stage process was followed to investigate (1) the basic internal assessment qualities of the measure, (2) the added-value of the subscales (i.e., alignment with organisational values, meaningful work, sense of community) to the model, and (3) the assessment of the external validity of the measure in relation to an external variable (in this case perceived employee performance). Different indices are consulted during the three stages to determine (1) whether the measurement in question is essentially unidimensional in nature and (2) whether the estimated group factor scores are better predictors of the criterion than the estimated general factor scores.Main findings: The various indices suggested that the measure of workplace spirituality could be treated as multidimensional and essentially unidimensional in nature. The three primary factors (alignment with organisational values, meaningful work and sense of community) resulted in a significant (yet small) increase in accuracy of predicting the estimated scores associated with perceived employee performance compared with the prediction based on the estimated scores for the general factor (workplace spirituality). However, the presence of a strong general factor cannot be ignored, pointing to a measure that is also essentially unidimensional and to be scored accordingly – allowing for quick and accurate assessment of individuals’ levels of workplace spirituality.Practical/managerial implications: Applied researchers and practitioners should take note of the theoretical and statistical value associated with the subdimensions of workplace spirituality to better understand why these dimensions are predictors of employee performance. The results of the study emphasise the important role of both personal-level and organisational-level variables associated with workplace spirituality in relation to perceived employee performance.Contribution/value-add: The study suggests that this measure of workplace spirituality can be treated as essentially unidimensional and multidimensional in nature – depending on the purpose of the assessment (fine-grained versus general). In addition, the present study provides evidence of the usefulness in employing novel procedures to determine the dimensionality of an instrument using external validity evidence.

What People Say and How People Play: Mixed Methods and Multiple Measures of Fishing Behavior with Implications for External Validity

Does resource extraction during experimental economic games reflect resource use and behavior in other contexts? This question about external validity is central to determining the inferences that can be drawn from experimental evidence to broader sets of circumstances. Building on previous studies of external validity, which often raise concerns about the generalizability of experimental economic games to other contexts, we instead highlight parallelism between behavior inside and outside of games and its implications for applied research. Here we present the results from a framed multiphase common-pool resource game played with fishermen from a fishing community in Baja California Sur to explore the associations between game play and both self-reported and observed behavioral measures. Participants played an experimental economic game that incorporated both fisheries-specific resource dynamics and the current policy context of local fisheries management. Game administration followed two years of interviews, participant observation, and household surveys in the community. In addition to providing an in-depth understanding of the broader social-ecological context, this allowed us to compare similar game, self-reported, and observed behaviors for the same individuals. Overall, we found little evidence for external validity in terms of parallelism. None of the tests for association between fishing behaviors in the game and survey met the study-wide significance threshold. Only one association—between game behavior and conservation values—was significant at the single-test level. Based on our results, we caution against relying solely on experimental economic games as proxies for behaviors regarding resource use when developing policies. Collecting multiple or iterative measures of behavior using mixed methods is a necessary precaution, especially if the goal is to make specific policy recommendations.

Work as Meaning Inventory: Propriedades Psicométricas e Evidências Adicionais da Versão Brasileira

Abstract Meaningful work promotes the professional development of individuals and organizations. This study aimed to translate, adapt, and present additional evidence of the validity of the Work as Meaning Inventory - WAMI for the Brazilian population. It addressed 2,111 workers (67% women) aged between 18 and 77. The results indicate that the inventory’s Brazilian version has a linguistic structure appropriate to the Brazilian Portuguese and satisfactory goodness of fit indices for the one-factor structure, in addition to invariance across sexes. Additionally, evidence of external convergent validity was found with life satisfaction and job satisfaction, which was negative for exhaustion. The conclusion is that meaningful work benefits the well being of workers and organizations. Thus, we recommend that this adapted version be used in the Brazilian context, as it meets the psychometric requirements for adapting instruments across cultures.

Development of the Chemistry Mindset Instrument (CheMI) for use with introductory undergraduate chemistry students

Chemistry education research has increasingly considered the role of affect when investigating chemistry learning environments over the past decade. Despite its popularity in educational spheres, mindset has been understudied from a chemistry-specific perspective. Mindset encompasses one's beliefs about the ability to change intelligence with effort and has been shown to be a domain-specific construct. For this reason, students’ mindset would be most relevant in chemistry if it were measured as a chemistry-specific construct. To date, no instrument has been developed for use in chemistry learning contexts. Here we present evidence supporting the development process and final product of a mindset instrument designed specifically for undergraduate chemistry students. The Chemistry Mindset Instrument (CheMI) was developed through an iterative design process requiring multiple implementations and revisions. We analyze the psychometric properties of CheMI data from a sample of introductory (general and organic) chemistry students enrolled in lecture courses. We achieved good data-model fit via confirmatory factor analysis and high reliability for the newly developed items, indicating that the instrument functions well with the target population. Significant correlations were observed for chemistry mindset with students’ self-efficacy, mastery goals, and course performance, providing external validity evidence for the construct measurement.